Systems and methods for intelligence delivery

ABSTRACT

A system and method for selecting tailored information to present to a subscriber, including receiving subscriber data, determining an analysis set based on the subscriber data, extracting abstract parameters from the subscriber data, selecting an analysis from the analysis set based on a general model and the abstract parameters, and presenting the selected analysis to the subscriber.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. application Ser. No.17/014,112, filed 8 Sep. 2020, which claims the benefit of U.S.Provisional Application No. 62/897,169, filed 6 Sep. 2019, both of whichare incorporated in their entirety by this reference.

TECHNICAL FIELD

This invention relates generally to the data-intensive complex computingarchitecture field, and more specifically to a new and useful system andmethod in the data-intensive complex computing architecture field.

BACKGROUND

In the normal course of operations, entities create considerable to verylarge amounts of electronic data resulting from their operations. Insome cases, the amount of electronic data generated can be in the tensof thousands to millions of units of data per day thereby resulting inextremely large data sets (e.g., big data), which can be unstructuredand structured. Using big data platforms, some of these entities seek toleverage their big data to obtain beneficial insights and this is done,mainly, by utilizing the big data platform to store the large volume ofdata and organize the data in a format that is searchable via queries.

A challenge with this model of using the big data platform, however, isthat in order to obtain the useful insights that the entities envisionsto obtain, analysts and other users of the big data platform must beable to run appropriate queries against the data in the platform. Thus,in such a model, the insights may only be useful if the queries againstthe data are good.

To assist in the use of big data platforms, some software applicationsare implemented in big data platforms to analyze the incoming data. Insuch instances, to determine useable data, these applications applysubstantial analysis against each unit of datum of incoming data,organize the data, and potentially run automated queries thereon toprovide insights or information to the administrator. However, analyzingeach unit of datum of these very large datasets in this manner usurpssignificant computing resources and in turn, delays the data processingand insight determination due to overuse of the computer processors,memory, and other technical computing elements of the big data platform.Further, there is no guarantee that the queries generated by thesoftware applications will, in fact, identify useable data and returnuseful insights.

Thus, there is a need in the data-intensive complex computingarchitecture field to create new and useful systems, methods, andapparatuses to be implemented in a data-intensive complex computingarchitecture for processing big data, identifying useful data, andgenerating meaningful and exploratory intelligence therefrom that mayfurther be based on machine learned subscriber preferences. Theembodiments of the present application provide such new and usefulsystems, methods, computer program products, and apparatuses.

BRIEF DESCRIPTION OF THE FIGURES

FIG. 1 illustrates a schematic representation of a system in accordancewith one or more embodiments of the present application.

FIG. 2 illustrates a schematic representation method in accordance withone or more embodiments of the present application.

FIG. 3 illustrates a schematic representation of an example ofgenerating a model.

FIGS. 4A and 4B illustrate schematic representations of a relationshipbetween a plurality of distinct subscribers and a general model and theplurality of subscribers and a specific model, respectively, inaccordance with one or more embodiments of the present application.

FIG. 5 illustrates exemplary interactions between a detection system,story generator, and feed manager through story features and modelfeatures in accordance with one or more embodiments of the presentapplication.

FIG. 6 is a schematic representation of a data flow in accordance withone or more embodiments of the present application.

FIG. 7 is a schematic representation of an example of a user interactionand the use of a user interaction to update or train a customizedgeneral model and a specific model.

FIG. 8 is a schematic representation of a first illustrative example ofa model used to generate an interaction score for an analysis.

FIG. 9 is a schematic representation of a second illustrative example ofa model used to select an analysis.

FIG. 10 is a schematic representation of a third illustrative example ofa model used to generate an interaction score for an analysis.

DESCRIPTION OF THE PREFERRED EMBODIMENTS

The following description of the preferred embodiments of the inventionis not intended to limit the invention to these preferred embodiments,but rather to enable any person skilled in the art to make and use thisinvention.

1. Overview.

As shown in FIG. 2, the method S200 can include receiving data S210,determining an analysis set S220, abstracting the data S230, selectinganalysis from the analysis set S240, and presenting the selectedanalysis S250. The method can optionally include generating a generalmodel S260, generating a customized general model S265, generating aspecific model S270, monitoring the model(s), and/or any steps.

As shown in FIG. 1, the system 100 can include a detection system 110, astory generator 120, a story feed manager 130, a datastore 140, anapplication programming interface (API) 150, a modelling system 160, amodel monitor (monitor) 170, and/or any components.

Embodiments of the system and method preferably function to select andpresent tailored information to one or more users. The tailoredinformation is preferably information (e.g., analyses of or associatedwith subscriber data) that the user(s) are anticipated to be more likelyto interact with, but can correspond to any information. For example, auser associated with a user class that favors a particular geographicalarea (for example, a sales manager working with the pacific northwestregion of America) might be more likely to be presented informationrelated to the states of Washington and Oregon than information relatingto other states. However, the tailored information can correspond to anyinformation.

In an illustrative example as shown in FIG. 6, the method can include:receiving data associated with an entity (e.g., subscriber); determininga set of stories (e.g., an analysis set) based on the data; extractingabstract parameters from the data; selecting a story from the set ofstories based on a general model and the abstract parameters; andpresenting the selected story to a user or user view associated with theentity. The abstract parameters are preferably anonymized such that thedata cannot be (or at least cannot easily be) associated with a givenentity. The general model is preferably generated from abstractparameters extracted from data associated with a plurality of entities,stories for each of the plurality of entities, and user interactionswith the stories for each of the plurality of entities. As more data anduser interactions are collected for a given entity, the method canoptionally include generating entity-specific selection models based onthe entity's data and the entity's story interactions, wherein theentity-specific models are used in lieu of, or in addition to, thegeneral model to select the story for entity presentation. However, themethod can include any suitable steps.

2. Benefits.

Variations of the technology can confer several benefits and/oradvantages.

First, the Applicant has discovered that many subscribers (e.g.,customers, accounts, entities, etc.) have similar interests. Therefore,using the experiences of one subscriber can facilitate enhanced analysisselection and presentation to other subscribers. As differentsubscribers typically do not (and do not want to) share data with oneanother, the Applicant has found that a challenge with doing this isensuring that each subscriber's data remains anonymous to othersubscribers. Applicant has discovered that selecting the stories topresent using abstracted parameters (e.g., by abstracting the data) canhelp ensure that data is more likely to remain anonymous to othersubscribers while also enabling improved selection of and presentationof analysis to one or more users.

However, variants of the technology can confer any other suitablebenefits and/or advantages.

3. System.

As shown in FIG. 1, an intelligence system 100 can includes a detectionsystem 110 (e.g., an anomaly detection system), a story generator 120, astory feed manager 130, a datastore 140, and an application programminginterface (API) 150, a modelling system 160, and a model monitor(monitor) 170. In specific embodiments, the intelligence system and/orcomponents thereof can operate in a manner as described in in U.S. Pat.No. 10,313,466, filed 30 Sep. 2016 entitled ‘SYSTEM, APPARATUS, ANDMETHOD TO IDENTIFY INTELLIGENCE USING A DATA PROCESSING PLATFORM’ orU.S. patent application Ser. No. 16/392,874 filed 24 Apr. 2019 entitled‘SYSTEM, APPARATUS, AND METHOD TO IDENTIFY INTELLIGENCE USING A DATAPROCESSING PLATFORM’, each of which is incorporated herein in itsentirety by this reference. The intelligence system and/or one or morecomponents of the intelligence system can be implemented in a platformdomain (e.g., a client operated by an intelligence service operator orprovider), a subscriber domain (e.g., a client controlled or hosted by asubscriber), distributed between a platform domain and a subscriberdomain, and/or in any suitable domain. The platform domain preferablytransmits the general model (e.g., a trained general model) to thesubscriber domain, but can additionally or alternatively transmit scores(e.g., associated with an analysis of the analysis set, wherein themodel input values can be determined at the subscriber domain andtransmitted to the platform domain), the analysis set, the customizedgeneral model, specific model, other models, abstracted data, data(e.g., public data, subscriber data, etc.), and/or any suitable contentto the subscriber domain. The subscriber domain can transmit subscriberdata, abstracted data, the specific model, the customized general model,scores, the analysis set, public data, no data, and/or any content tothe platform domain.

The intelligence system is preferably shared across multiple subscribers(e.g., wherein all subscriber data is analyzed by the same instances ofdifferent modules); alternatively, a different intelligence systeminstance can be used for each subscriber (e.g., to promote proprietarydata security), or specific modules can be replicated for eachsubscriber while other module instances are shared (e.g., datastores 140and story generators 120 are replicated, but the story feed manager 130,modelling system 160, and model monitor 170 are shared). However, theintelligence system can be otherwise allocated to different entities.

In one or more embodiments, the intelligence system 100 may beimplemented by a distributed network of computers (e.g., the cloud orthe like) and/or by one or more computer processing circuits and thelike. Preferably, the intelligence system 100 may be a cloud-basedsystem. However, the intelligence system can additionally oralternatively be implemented on a local computing system, implemented inan edge computing system, and/or be implemented on any suitablecomputing systems. Additionally, or alternatively, the intelligencesystem 100 may enable the implementation of an intelligence service thatprovides intelligence services to one or more subscribers (e.g.,subscribers, entities, etc.) to the intelligence service having one ormore accounts with the service. Accordingly, the intelligence system 100may sometimes be referred to herein as the intelligence service 100.

In one or more embodiments, the modelling system 160 (e.g., machinelearning system) may function to generate and/or deploy one or moremodels (e.g., machine learning models) including: a general model 161(e.g., a global machine learning model, global probability distributionmodel), a customized general model 162, a specific model 163 (e.g.,specific machine learning model, specific probability distributionmodel), and/or any suitable models. As shown for example in FIG. 5, themodelling system is preferably communicably coupled to the feed managerand the monitor, but can additionally or alternatively be coupled to adatastore, story generator, detection system, API, and/or anycomponents. The models can be generated, trained, and/or updated once,with a predetermined timing (e.g., every day, week, month, quarter,year, etc.), responsive to a trigger (e.g., a model update trigger, athreshold amount of data collection, threshold number of userinteractions with stories, a change in a market, etc.), randomly, basedon a monitor of the general model, and/or with any timing.

The models, as implemented by the modelling system 160, may include asingular model and/or an ensemble of predictive models dedicated toinforming an operation of the story feed manager 130 for one or moresubscribers to the intelligence service 100 (for example as shown inFIG. 4A). Examples of operations of the story feed manager can include:which stories to present, an ordering for stories to presenter, storyclasses to present, order for story classes to present, and/or anysuitable operation of the story feed manager, wherein the story feedmanager can select, order, present, or otherwise operate based on themodel outputs (e.g., scores per story or story class).

The general model 161 is preferably shared across multiple subscribers,and functions to guide story selection for each of a plurality of users(e.g., can be used to select stories for all or substantially allsubscribers). The system can include a shared general model 161 for: allor substantially all subscribers to the intelligence service; for eachgroup of subscribers that share common parameters (e.g., same businessvertical, same industry, same geographic location, same revenuestructure, same subscriber class, same view type, etc.); and/or anyother suitable set of subscribers.

The general model is preferably generated (e.g., trained) and/orinitialized using data, preferably abstracted data, associated with aplurality of subscribers, the analysis set corresponding to theplurality of subscribers (e.g., subscribers' stories or featuresthereof), and the interactions of the plurality of subscribers withtheir respective analysis set. When the system includes general modelsspecific to a subscriber group, the general model can be generated basedon the data, stories, and respective user interactions from subscriberswithin the subscriber group. However, the general model can be otherwisegenerated.

The general model preferably accepts abstract data as an input (e.g.,model feature(s) 167), and does not accept (proprietary) data as inputs.However, the general model can additionally or alternatively acceptproprietary data, story features (e.g., feature values from the specificstory), and/or other information as an input. The general model canoutput one or more: analysis selections (e.g., selects a story),analysis scores (e.g., score for each story), information classselections (e.g., selects a story type), analysis class scores (e.g.,score for the story type), a new analysis (e.g., a new story), or anyother suitable output. When the general model is used to select a storyfor a given subscriber, the abstract data is preferably generated solelyfrom the subscriber's proprietary data, but can optionally includeabstract data from other sources (e.g., public sources, othersubscribers' data).

In a preferred embodiment, the modelling system can update and/orcustomize the general model to generate a customized general model. Thecustomized general model is preferably subscriber specific (e.g., one ormore different customized general models are generated for eachsubscriber), but can be customized for a user class, subscriber class,and/or for any suitable user(s). The customized general model can begenerated by training or updating the general model using data (e.g.,abstracted data, analysis set, interactions, etc.) for a singlesubscriber. However, the customized general model can be otherwisegenerated and/or used.

The customized general model preferably only accepts abstract data as aninput, but can additionally or alternatively accept proprietary data,story features 122, or other data. The customized general model canoutput one or more: analysis selections or scores, analysis classselections or scores, new analyses, or any other suitable output.

In a preferred embodiment, the modelling system can generate and/orupdate a specific model. The specific model is preferably subscriberspecific (e.g., one or more different specific models are generated foreach subscriber, for example as shown in FIG. 4B), but can be customizedfor a user class, subscriber class, and/or for any suitable user set.The specific model can be generated by training a model using data(e.g., subscriber data, analysis set, interactions, etc.) for a singlesubscriber. When the specific model is specific to a user subset (e.g.,user class, view class, etc.), the specific model can be generated basedon the subscriber's data and the user subset's interactions. However,the specific model can be otherwise generated and/or used.

The specific model preferably accepts proprietary data as an input, butcan additionally or alternatively accept abstract data, story features,or other data. The specific model can output one or more: analysisselections or scores, analysis class selections or scores, new analyses,or any other suitable output. The data used by the specific model ispreferably not used by the general model (or customized general model);however, the specific model and general model (or customized generalmodel) can share some or all data. In an illustrative example, theabstract parameters used by the general model to select an analysis ofthe analysis set are not extracted from the subscriber data used by thespecific model to select the analysis of the analysis set. In a secondillustrative example, the abstract data used by the general model can beextracted from the subscriber data used by the specific model, whereinthe abstract data's variables are different from the subscriber data'svariables.

The monitoring module 170 may function to monitor the modelling system160 and/or the one or more models deployed by the modelling systemincluding, for example, the general model 161, the customized generalmodel 162, and/or the specific model 163. In some embodiments, themonitoring module can correspond to an intelligence system such asdisclosed in U.S. patent application Ser. No. 16/392,874 or in U.S. Pat.No. 10,313,466. However, any monitoring module can be used.

The system can be used with data. The data is preferably associated withthe subscriber (subscriber data), and is preferably proprietary orconfidential to the subscriber (e.g., accessible to the subscriber, suchas via a login, but not accessible to other subscribers). Alternatively,the data can be publicly accessible or otherwise accessible. The datacan include values for one or more data attributes (e.g., dataparameter, data characteristic, data variable, etc.), can be associatedwith metadata (e.g., with metadata values for each metadata attribute),and/or can be associated with other information. Examples of datainclude: sales records (e.g., with $100, 3 Jul. 2019, and Richmond,Calif. as the values for the sales price, date, and location attributes,respectively), marketing data, manufacturing data, or other data(further examples described below). Examples of metadata include:timestamp, location, subscriber identifier, change over time, or othermetadata.

The system can be used with abstracted data (abstractions, derived data,abstract parameters, etc.). The abstracted data can be calculated,extracted, or otherwise determined from the (proprietary) data. Theabstracted data can be determined by the story generator, the datastore, a data extractor (e.g., dedicated to the abstracted data variableor feature; a multi-feature extractor; etc.), and/or other module. Theabstracted data can include values associated with an abstracted dataclass (e.g., type, variable, etc.), metadata (e.g., source dataidentifiers, subscriber identifier, etc.), and/or other information.Examples of abstracted data include: metadata variables, metadatavalues, dataset characterization (e.g., mean, median, mode, standarddeviation, etc.), relationship to the dataset (e.g., deviation, cluster,etc.), data attributes (e.g., the attribute itself, not the associatedvalue), and/or other information.

The system is preferably used by a set of subscribers (e.g., companies,entities, customers, accounts, master accounts, etc.). Each subscribercan be associated with one or more user views or user classes (e.g.,user group, such as marketing or sales). Each user class can be accessedby one or more users (e.g., individuals). Each user can be associatedwith authentication credentials (e.g., login information, 2-factorauthentication, etc.) that enables the user to access stories (andoptionally data) proprietary to the respective user class or subscriber.However, the system can be used by any other suitable set of users.

4. Method.

A method for providing tailored information can include receiving dataS210, determining an analysis set S220, abstracting the data S230,selecting an analysis from the analysis set S240, and presenting theselected analysis S250. The method can optionally include generating ageneral model S260, generating a customized general model S265,generating a specific model S270, monitor one or more model(s), and/orany steps. The method can be implemented by an intelligence system(e.g., as described above) and/or by any suitable system. The methodand/or one or more steps thereof can be performed on a platform client(e.g., a client maintained by an intelligence system operator), asubscriber client (e.g., a client controlled or operated by asubscriber), distributed between a platform client and a subscriberclient, and/or at any suitable client(s).

The method preferably functions to select and/or organize analyses froman analysis set to present to one or more users. The tailored (e.g.,selected, organized, etc.) information is preferably one or moreanalysis from the analysis set that the user(s) are predicted to be morelikely to interact with (e.g., based on interactions of the user withprevious information). However, the tailored information can be newanalyses synthesized from the analysis set, analyses that are predictedto exceed a metric (e.g., importance, impact, uniqueness, unexpected,value, etc. metrics), and/or any suitable information. The methodpreferably generates unique tailored information for each subscriber(e.g., an entity which subscribes to the platform), subscriber class(e.g., a geographic class such as associated with a particulargeographic region; a subscriber size; a subscriber field of work; etc.),user class (e.g., marketing, sales, management, technical development,human resources, legal, other teams or subteams associated with one ormore department or division, etc.), one or more individuals, and/or withany information recipient. However, the tailored information may be thesame for different information recipients and/or information recipientclass.

One or more instance of the method and/or steps thereof can be performedsimultaneously and/or sequentially. Each instance of the method can use(e.g., process) the same or different sets of data. Each instance of themethod can be performed in the same domain (e.g., at the same client) orin different domains (e.g., on different clients).

Receiving data S210 (e.g., generating a first or second corpus of data)preferably functions to collect data associated with and/or of interestto a subscriber. In some embodiments, the data corresponds to anomalousdata (e.g., outliers, inflection points, changes, etc.) of thesubscriber data. However, the data can correspond to any data. Someexamples of subscriber data include: financial data, sales data,marketing data, social media data (e.g., likes, retweets, comments,shares, reviews, etc.), personnel data (e.g., number of sick days used,number of employees taking sick days, number of vacation days used,number of vacation days remaining, new hires, new offboarding events,firing, etc.), technology usage data (e.g., amount of time spent usingtool such as capital equipment, software, etc.; number of users using atool; etc.), competitor data, stock market data, government data (e.g.,new laws passed, changes to tax code, etc.), investor data, and/or anysuitable subscriber data. The data is preferably collected by thesubscriber, but can additionally or alternatively be collected by aservice provider (e.g., the intelligence system provider, a datacollection agency, etc.), a competitor, another subscriber, a governmentagency, a legal agency, an investment agency, and/or by any entity. Thecollected data can be actively or passively generated. The collecteddata is preferably stored by the datastore 140, but can be otherwisestored. The collected data is preferably stored in a separate datarepository from other subscribers' data, but can be stored with othersubscribers' data or otherwise stored.

S210 is preferably performed at a subscriber client, but canadditionally or alternatively be performed at a platform client and/orat any suitable client. S210 is preferably performed by a detectionsystem, but can additionally or alternatively be performed by adatastore, a story feed manager, a modelling system, a model monitor,and/or by any suitable component.

Generating an analysis set S220 preferably functions to determine a setof information about or based on the data (and/or abstracted data asgenerated for example in S230). The analyses within the analysis setpreferably includes values for each of a set of analysis classes, butcan be otherwise characterized. An analysis class can be: an analysistype, story type, observation type, or other analysis class. Examples ofanalysis classes can include: spike and drop, relationships, trends,milestones, funnels, and/or other classes. The information value can be:data values cooperatively forming the story, data values cooperativelyforming the observation, and/or other values. Examples of informationvalues include: a subscriber-specific metric increase or decrease (e.g.,story of the spike-and-drop class), a relationship between differentareas of a subscriber's business (e.g., story of the relationshipclass), a trend in the subscriber's data (e.g., story of the trendclass), and/or other analyses.

Information for a subscriber is preferably generated from the datavalues associated with the subscriber (e.g., proprietary data values),but can additionally or alternatively be generated from the abstracteddata values generated from the proprietary data, public data, and/orother data.

The analyses in the analysis set can include one or more: stories,unexpected observations (e.g., an outlier), interesting data points(e.g., exceeding a threshold value, exceeding a threshold change invalue, a sustained increase relative to a threshold, failing to achievea threshold value, failing to achieve a threshold change in value, asustained decrease relative to a threshold, etc.), correlations in thedata, trends in the data, a change in a characteristic of the data(e.g., order of magnitude change, change in the variance, change in theskewness, etc.), insights (e.g., as described in U.S. Pat. No.10,313,466), and/or any information. In an illustrative example, theanalysis can include one or more questions that are raised by the data(e.g., a question and/or suggestion for a topic to look in to further).However, the analysis can answer a question, format the data (e.g.,graphically), process the data (e.g., extract signals from noise),and/or otherwise be generated from or related to the data.

The analysis set is preferably generated at a subscriber client, but canadditionally or alternatively be generated at a platform client and/orany suitable client. The analysis set is preferably generated by a storygenerator, but can additionally or alternatively be generated by anycomponent of the intelligence system or be otherwise generated.

The analysis set (and/or one or more analysis contained therewithin) ispreferably associated with a time span (e.g., time duration). The timespan can correspond to: the amount of time that the analysis is(predicted to be) relevant, the amount of time that the analysis is(predicted to be) interesting, the amount time before the analysis (ispredicted to) changes, the time the data was acquired, the time the datawas processed, a time stamp associated with the data, the time the datais (presumed) valid, a target time (e.g., determined by a subscriber,determined by a platform provider, determined by a data collectionservice, etc.), a predetermined amount of time (e.g., based on acharacteristic of the analyses), a random amount of time, a subscriberpreference, based on the user view, a time window associated with astory class, and/or any suitable time span. Examples of time spansinclude: an hour, 2 hours, 4 hours, half a day (e.g., half a work day),6 hours, 8 hours, 10 hours, 12 hours, 15 hours, 24 hours, 1 work day, 1day, 2 days, 3 days, 4 days, 5 days, 6 days, 7 days, 10 days, afortnight, a month, two months, a quarter, six months, a year, twoyears, 5 years, a decade, a score, a century, greater than a century,less than an hour, and/or any amount of time therebetween.

In a specific example, the analyses can be generated in the same or asimilar manner as identifying intelligence information and insights asdisclosed in U.S. Pat. No. 10,313,466 or U.S. patent application Ser.No. 16/392,874. However, the analysis set can be generated in anymanner.

Abstracting data S230 (e.g., generating a corpus of data) functions toanonymize the data and/or information so that the data source (e.g., thesubscriber) and/or data values are concealed. The data can be abstractedby the feed manager, the datastore, the story generator, the detectionsystem, the modelling system, the monitoring system, and/or by anycomponent of the intelligence system or by any system. S230 ispreferably performed in the subscriber domain (e.g., at a subscriberclient), but can additionally or alternatively be performed in theplatform domain and/or in any suitable domain. The abstracted data(e.g., abstract parameters) can correspond to, be extracted from, and/orbe generated from the subscriber data, the analyses, data extracted fromthe analyses (e.g., titles, length, etc.), and/or any suitable data.

Examples of abstracting the data can include one or more of:transforming the data (e.g., rounding; Fourier transformation;convolution; mathematical operations such as addition, multiplication,subtraction, division, etc.; etc.), removing or obscuring one or moreaspect of the data (e.g., removing a data label, randomizing a datalabel, randomizing a data value, etc.), data suppression, datageneralization, encryption, data randomization or pseudorandomization,normalization (e.g., normalized based on time span, number of employees,number of subscribers, subscriber class, number of vendors, number ofconsumers, etc.), and/or otherwise abstracting the data. In anillustrative example, numerical data can be abstracted by determining achange (e.g., in magnitude, in order of magnitude, sign such as increaseor decrease, percent change, etc.) in the data value (e.g., relative toan adjacent data point, relative to the same data point at a differenttime, etc.). In a second illustrative example, abstracted data cancorrespond to determining a bin the data value fits in such as an orderof magnitude (e.g., rounding the data value to the nearest order ofmagnitude), a predetermined bin, logarithmic bin (e.g., relative to anysuitable base), and/or otherwise.

In some variants of the invention, the abstracted data can be associatedwith one or more shape, facet, and/or attribute of the set of dataand/or the analyses. In related variants, the abstracted data can beassociated with and/or include metadata of or about the data.

In an illustrative example, the abstracted (e.g., derived) data isderived from, but does not explicitly include, the data (e.g., does notexplicitly include the data values, data types, data format, etc.).However, the abstracted data can include (e.g., explicitly, implicitly)the data.

Selecting one or more analysis from the analysis set S240 functions toselect a subset of information to display to one or more viewers (e.g.,users, user class, subscriber, etc.), but can alternatively function toselect the ordering for the information (e.g., informationprioritization), or otherwise influence information presentation to theuser. The selected analysis is preferably tailored information (e.g.,information which the viewer would like to view, needs to view, is morelikely to interact with, etc.), but can be any or all of the informationfrom the analysis set. In a preferred embodiment, the information istailored for a user class (e.g., a team, division, department, group,etc. of, within, or associated with the subscriber). A subscriber can beassociated with any number of user classes and as such can have anynumber of tailored information views. Examples of user classes caninclude: management, marketing, technical development, software,hardware, legal, human resources, information technology, operationsmanagement, finance, and/or any suitable user classes. However, theinformation can be tailored for a user, a subscriber, and/or anysuitable viewers.

The selected analysis is preferably selected by a feed manager 130and/or a modelling system, but can additionally or alternatively beselected by any component of an intelligence system and/or any suitablesystem. The information is preferably selected at a subscriber client,but can additionally or alternatively be selected at a platform clientand/or any suitable client.

The selected analysis is preferably selected using one or more models.The models can be general models, customized general models, subscriberspecific models, and/or any suitable model(s). Each model that is usedcan be general, subscriber specific, user class specific, geographicregion specific, time span specific, user specific, subscriber classspecific, subscriber group specific (e.g., used for a plurality ofsubscribers), and/or be applicable to any set or subset of viewers orsubscribers. However, the selected analysis can additionally oralternatively be selected according to subscriber preferences, userpreferences, time (e.g., time of creation, time since information waslast viewed, etc.), information content (e.g., information type), and/orbased on any suitable characteristics of the information. Generalmodels, including customized general models, preferably use abstracteddata as the input, but can use any suitable data as inputs. Specificmodels preferably use subscriber data (e.g., including confidentialdata) as inputs, but can additionally or alternatively use abstracteddata or any suitable data.

When a plurality of models is used, the inputs to each model arepreferably not identical to nor derived from the same source. In anillustrative example, the inputs to a general model (or customizedgeneral model) correspond to abstracted data and the inputs to aspecific model correspond to subscriber data (e.g., not abstracteddata). In this illustrative example, the subscriber data processed bythe specific model is preferably not used to generate the abstracteddata processed by the general model. However, the inputs to each modelmay be the same, derived from the same data, and/or use any suitabledata.

Each model is preferably trained (e.g., generated) using a training setof data. The training data preferably correspond to a training analysisset and a corresponding set of subscriber interactions to the traininganalysis set. The training data for the general model is preferablygenerated by a plurality (or all) subscribers. The training data for thespecific model and the customized general model are preferably generatedby a specific subscriber. However, any model can be trained using anytraining data.

One or more models preferably corresponds to a set of probabilitydistributions (e.g., a pre-determined set of learned probabilitydistributions), but can additionally of alternatively correspond to aneural network, a nearest neighbor approach, a decision tree or decisionforest, a set of rules and/or heuristics, and/or any suitable modeltype. When a plurality of models is used, each model is preferably, butdoes not have to be, the same type of model. In an illustrative example,training the models can correspond to determining probabilitydistribution shapes, weights, relationships, and/or otherwise correspondto determining a relationship between model parameters, data inputs, anda score associated with an analysis of the analysis set. The probabilitydistributions of the set of probability distributions can be representedgraphically (e.g., histogram), by a table, by an equation (e.g.,probability density function, probability mass function, cumulativedistribution function, etc.), and/or be otherwise represented.

In a specific example (example shown in FIG. 8), each model can includea set of probability distributions, each associated with a differentdata variable (e.g., abstract data variable or model feature; subscriberdata variable; etc.). The probability distribution for a data variablecan encode interaction frequencies or probabilities (e.g., determinedfrom historical interaction frequencies) for different values of thedata variable, and can optionally encode noninteraction frequencies fordifferent values of the data variable. The data variable can becontinuous or binary; the values for continuous features can besegmented into bins (e.g., predetermined increments, percentiles, etc.)or otherwise managed. The data variables included in the model (e.g.,data variables for which probability distributions are generated) canbe: automatically determined (e.g., using another model that identifiesthe most influential data variables), manually determined, and/orotherwise determined. The timeframe encompassed by each probabilitydistribution is preferably the same for probability distributions withinthe same model, but can alternatively be different. The timeframes canbe manually determined, automatically determined (e.g., based on when athreshold amount of data available, when the data stabilizes and hasless than a threshold proportion of outliers, etc.), and/or otherwisedetermined. Alternatively, the model can include: a single probabilitydistribution encoding interaction and/or noninteraction frequencies witheach of a plurality of values for each of a plurality of data variables;or be otherwise constructed.

Different models of the same type can include probability distributionsfor the same features (different variables), but differ in theunderlying data (and therefore, interaction probability per variablevalue). Alternatively, different models of the same type can includeprobability distributions for different features (data variables),probability distributions for the same number of features (e.g., suchthat the scores from individual probability distributions do not have tobe weighted), and/or be otherwise related. Different models of differenttypes (e.g., general vs. specific) can have different features (e.g.,data variables), but can share the same number of variables or beotherwise distinct or related.

In a second illustrative example (example shown in FIG. 9), each modelcan include a multi-class classifier that outputs an analysis classselection, or probabilities for each of the plurality of analysisclasses.

In a third illustrative example, each model can include a set ofsingle-class sub-models or classifiers (e.g., one for each analysisclass, one for each feature, etc.) that outputs an interactionprobability. When the classifiers output probabilities for an analysisclass, the analysis class with the highest probability can be selected.When the classifiers output interaction probabilities based on a featurevalue (example shown in FIG. 10), the individual probabilities can becombined (e.g., summed) to generate a score for each analysis, whereinthe analysis with the highest score(s) can be selected for presentation.

However, the models can be otherwise constructed.

Each model that is used in selecting the analysis preferably generates ascore. Each analysis of the analysis set is preferably associated with ascore (e.g., probability of interaction, given the data variable'svalue, based on historical interaction patterns with stories derivedfrom data having the data variable's value). However, additionally oralternatively, the score can be associated with a particular ordering ofthe analysis set (e.g., a score for each order index-analysiscombination), the analysis set, the data and/or abstracted data, asubset of the analysis set, and/or with any suitable content. Theselected analysis preferably corresponds to the analysis associated withthe highest score. However, the selected analysis can correspond to anyanalysis(es) from the analysis set with a score greater than athreshold, random analysis(es) from the analysis set (e.g., selectedrandomly from a curated subset of the analysis set such as analyses witha threshold score, randomly selected when two or more analyses have thesame score, etc.), an ordering of the analysis set (e.g., organizing theanalysis set from the highest scoring analysis to the lowest scoringanalysis), and/or otherwise be selected based on the score.

In variants when two or more models are used to select an analysis ofthe analysis set, the analysis can be selected based on the overallscore (e.g., the sum of the score generated by each of the models), theaverage score, the median score, a weighted score, the highest valuescore (of the set of scores), the lowest value score (of the set ofscores), a combined score (e.g., a weighted score), voting between themodels, based on the types of models used, and/or otherwise selectedbased on the outputs of the models.

Each score is preferably associated with a probability that the viewer(e.g., subscriber, user class, etc.) will interact (e.g., positivelyinteract with, negatively interact with, etc.) the tailored information(e.g., selected analysis or analyses). In an illustrative example ofthis embodiment, when the model corresponds to a set of probabilitydistributions, the interaction probability can be derived from (e.g.,calculated, cross-correlated, selected from a chart, determined using aneural network, etc.) the probability distributions. However,additionally or alternatively, the score can be associated with aquality of the analysis (e.g., a confidence that the analysis isinteresting), an impact of the analysis (e.g., more likely to select ananalysis that is predicted to have a large impact, where the model or analternative model additionally predicts an impact of the analysis), averacity of the analysis (e.g., a confidence that the analysis iscorrect), and/or associated with any metric of the analysis or analysisset.

In an illustrative example, S240 may deploy the general model and thespecific model in parallel. In this illustrative example, the generalmodel may be deployed in an online (live) state in which inferences andpredictions are generated and may be used to inform an operation of astory feed component of the service. The specific model may be deployedin an offline (non-live) state or training mode (e.g., as described inmore detail below), which may include a training phase for the specificmachine learning model and/or a validation phase. Accordingly,predictions and/or inferences of the specific machine learning in anoffline state may not be used to inform an operation of the story feedcomponent of the service.

Additionally or alternatively, S240 may include switching the specificmodel from an offline state to an online state based on one or moretraining metrics and/or performance metrics of the specific model. Thiscan be performed automatically, manually, or otherwise performed. Thatis, in one or more embodiments, S240 may evaluate one or more metrics ofthe specific model relative to one or more training and/or performancethresholds. Based on a result of the assessment of a readiness of thespecific model, S240 may function to (automatically) switch the specificmodel from an offline state to an online state. In some embodiments, forinstance when the specific model has achieved a threshold training orperformance metric (that can be the same as of different from thethreshold to switch the specific model on), S240 can optionally switchthe general model from the online state to an offline state.

Presenting the selected analysis, as recited in Block S250, functions topresent tailored information (e.g., the analysis selected in S240) fromthe analysis set to the subscriber and/or user. The tailored informationcan be presented as part of a story feed (e.g., as disclosed in U.S.Pat. No. 10,313,466), at a subscriber client, at a platform client, on auser device (e.g., using an application running on the user device suchas a smart phone, smart watch, laptop, personal computer, smart glasses,etc.), and/or otherwise present the information to the subscriber. S250can include presenting the tailored information, the analysis set (e.g.,organized such as according to the score, randomly, as generated, etc.),and/or any suitable information.

Presenting the selected analysis can include detecting one or moreinteractions with the selected analysis, which functions to determinehow the subscriber(s) interact with the information particularly but notexclusively the tailored information. The interactions are preferablydetected in the subscriber domain, but can be detected in the platformdomain and/or in any domain. The interactions are preferably transmittedto (and stored and/or processed by) the platform domain, but can betransmitted to, stored and/or processed in the subscriber domain and/orany suitable domain.

The interactions are preferably used (e.g., in conjunction with theselected stories) to train (and/or otherwise improve) the models toimprove the selection of tailored information (e.g., to better selecttailored information that the subscriber will interact with, for exampleas shown in FIG. 7). In some variants, interactions can be classified aspositive interactions, negative interactions, and/or neutralinteractions. However, the interactions can be unclassified and/orclassified in any manner. Generally, although not exclusively, positiveinteractions are associated with analyses that should be selected morefrequently while negative interactions are associated with analyses thatshould be presented less frequently. However, any interactions cancorrespond to analyses that should be presented more or less frequently.Examples of interactions can include: clicking (on a story), selecting,liking (or disliking), voting on (e.g., upvoting, down voting),commenting on, downloads, swiping, reading, browsing, dismissing,reacting to, sharing, an amount of time the analysis remains presented,and/or any suitable interactions. Any interactions can be used to trainor otherwise improve the models. Different interactions (e.g., types ofinteractions) can have the same or different weights for training themodel(s).

Generating a customized general model functions to train the generalmodel to better tailor information for a specific subscriber (and/or setof subscribers). Generating a customized general model can be performedby a modelling system and/or any component of an intelligence system orother suitable system. Generating a customized general model ispreferably performed in the subscriber domain, but can be performed inthe platform domain and/or in any suitable domain.

The customized general model is preferably generated based oninteractions of one or more users (preferably, but not exclusively, ofthe same user class) associated with the subscriber and the presentedanalysis. The customized general model can also be generated based onthe data (and/or abstract data) used to generate the information thatwas interacted with, and/or the data (and/or abstract data) used togenerate the analysis(es) (e.g., presented and/or not presented) thatwas not interacted with. However, the customized general model canadditionally or alternatively be generated based on a meta-analysis ofthe analyses, using unsupervised learning, and/or otherwise be updated.

The customized general model can be updated (e.g., trained or retrained)according to a schedule, with a given frequency (e.g., every day, everyweek, every fortnight, every month, every quarter, every half-year,every year, etc.), responsive to a change (e.g., a change in the model,a change in an entity structure, a change in the number or types of userclasses, etc.), randomly, responsive to a trigger (e.g., a request toupdate the model, receipt of a user interaction), and/or with anytiming.

In some embodiments, the customized general model can replace thegeneral model for generating the tailored information (e.g., for theuser class, for the subscriber, for a subscriber group, for allsubscribers, etc.). However, the customized general model and thegeneral model can each be used to generate the tailored informationand/or the customized general model and the general model can be used inany manner. For example, the general model can be used to monitor thecustomized general model.

Generating a specific model functions to generate and/or update asubscriber specific model. In one or more embodiments, the specificmodel is only accessible to a single subscriber. In these embodiments,the specific model is able to provide tailored information based onsubscriber data (e.g., proprietary data). However, the specific modelmay be generated to be specific for a given user class, a given marketsector, and/or other specific subscriber characteristics. The specificmodel is preferably generated in the subscriber domain (e.g., at asubscriber client), but can be generated in the platform domain (e.g.,at a platform client) and/or in any suitable domain. The specific modelis preferably generated by a modelling system, but can be generated byany component of an intelligence system or by any suitable system.

In a specific example, the specific model can be generated (or updated)by acquiring a training data set that includes subscriber data,presented information, and subscriber interactions with the presentedinformation; and training the specific model to select an analysis thatthe subscriber is more likely to interact with and/or to deprioritize(or not select) analyses that the subscriber is less likely to interactwith. However, the specific model can be trained in any manner.

As shown for example in FIG. 3, monitoring the models (e.g., the machinelearning models) functions to monitor an operation of the models (e.g.,the general model, the customized general model, the specific model,etc.). Additionally, or alternatively, monitoring the models mayfunction to identify or compute one or more performance metrics of themodel and assess a performance of the model based on the performancemetrics.

In one or more embodiments, monitoring the models may function tomonitor the model(s) for drift. In such embodiments, monitoring themodels can include computing a drift metric based on a calculateddifference between the predictions of the model and an assessedinteraction of a subscriber with one or more services of the service forwhich the predictions are made. However, the drift metric canadditionally or alternatively be derived based on the a comparisonbetween tailored information and/or subscriber interactions withanalyses selected using a first model (such as a general model,customized general model, specific model) and a second model (such as ageneral model, customized general model, specific model, the same as ordistinct from the first model, etc.), and/or otherwise be derived. Forinstance, if the model predicts that a subscriber may interact with apredicted tailored information (e.g., set of computer-generated stories)but the subscriber does not interact with or minimally interacts withthe predicted set of stories, monitoring the models may compute a driftvalue sufficient to potentially indicate a poor performance of themodel. Conversely, if a subscriber sufficiently interacts with thepredicted set of stories, monitoring the models may function tocalculate a low or minimal drift value. In such embodiments, monitoringthe models may function to assess the computed drift value against apredetermined and/or dynamic drift threshold such that if the computeddrift value does not satisfy and/or is below the drift threshold,monitoring the models may indicate that a performance of the model maybe adequate.

It shall be noted that while monitoring the models preferably monitorsthe specific model, monitoring the models may additionally oralternatively monitor the general model, the customized general model,and/or any model in a similar manner.

It shall be noted that monitoring the models may measure and/or monitorany suitable performance metric including, but not limited to, apredictive accuracy metric, a predictive efficiency metric, and thelike.

Additionally, or alternatively, monitoring the models may adjust orchange a state of the model based on monitoring one or more performancemetrics of the model. Specifically, in one or more embodiments,monitoring the models may function to switch an online/offline state ofthe model based on the one or more performance metrics of the model.

In a first implementation, monitoring the models may function to switcha state of the specific model from offline to online based on one ormore performance and/or operational metrics. For instance, in oneembodiment, monitoring the models may function to compute a trainingmetric for the specific model indicating a level of training that thespecific model has undergone. In such embodiment, monitoring the modelmay assess the training metric for the specific model against a trainingthreshold. In the circumstance that the training metric satisfies and/orexceeds the training threshold, monitoring the models may function toautomatically switch a state of the specific machine learning model fromoffline to online.

Additionally, or alternatively, monitoring the models may additionallyswitch a related but distinct general model from online to offline for aspecific subscriber for which the specific model was activated.Alternatively, monitoring the model may function to maintain an onlinestatus of the general model but only inform a story feed component ofthe service with the predictions and/or inferences of the specificmodel.

Additionally, or alternatively, as discussed above, monitoring the modelmay function to monitor and/or assess one or more performance metrics ofan online model. In one example, monitoring the model may function tocompute a drift value based on a performance of the model. In suchexample, monitoring the model may function to assess the drift value forthe model against a drift threshold. If the drift value meets or exceedsthe drift threshold (indicating a poor performance), monitoring themodel may function to automatically switch a state of the model fromonline to offline. Contemporaneously, or simultaneously, monitoring themodel may switch a state of a second model from offline/inactive toonline/active, such that the predictions of the second model inform anoperation of a story feed component of the service in lieu of the model.

In an illustrative example, monitoring the models can be performed by anintelligence service such as disclosed in U.S. Pat. No. 10,313,466,where the insights and information can correspond to the behavior and/orperformance of one or more models. However, monitoring the models can beotherwise performed.

Embodiments of the system and/or method can include every combinationand permutation of the various system components and the various methodprocesses, wherein one or more instances of the method and/or processesdescribed herein can be performed asynchronously (e.g., sequentially),concurrently (e.g., in parallel), or in any other suitable order byand/or using one or more instances of the systems, elements, and/orentities described herein.

As a person skilled in the art will recognize from the previous detaileddescription and from the figures and claims, modifications and changescan be made to the preferred embodiments of the invention withoutdeparting from the scope of this invention defined in the followingclaims.

We claim:
 1. A method for presenting a story to a subscriber comprising:receiving data associated with the subscriber; determining a set ofstories based on the subscriber data; extracting abstract parametersassociated with the subscriber data; selecting a story from the set ofstories based on a general model and the abstract parameters, whereinthe general model is generated from: abstracted parameters extractedfrom data from a plurality of subscribers, wherein the plurality ofsubscribers does not share data; stories for each of the plurality ofsubscribers; and user interactions with the stories for each of theplurality of subscribers; and presenting the story to a user viewassociated with the subscriber.